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1  INTRODUCTION 

The  Command  Center  Network  (CCN)  is  an  exploratory 
development  program  sponsored  by  the  Naval  Electronic  Systems 
Command.  The  CCN  provides  a  high-speed  local  area  network  to 
interconnect  a  variety  of  naval  command  center  resources  that 
previously  have  functioned  as  separate,  distinct  systems.  With 
the  interconnection  of  these  systems,  it  becomes  possible  for  the 
Navy  to  explore  the  establishment  of  standard,  higher-level 
protocols  to  support  exchange  of  information  among  diverse  users 
whose  systems  incorporate  varying  interfaces  and  communication 
protocols. 

Bolt  Beranek  and  Newman  Inc.  (BBN)  designed  and  implemented 
a  "communications  substrate"  for  the  Command  Center  Network. 
This  communications  substrate  consists  of  an  adaptation  of  a 
commercially-available  local  area  network  product,  LSI-11 
"front-end"  processors,  LSI-11  Gateway  and  Terminal  Interface 
Unit  processors,  and  a  UNIX-based  Network  Services  Manager.  BBN 
installed  the  communications  substrate  at  the  Naval  Ocean  Systems 
Center  (NOSC)  in  San  Diego,  California,  in  June  of  1981. 

BBN’s  participation  in  the  development  of  the  Command  Center 
Network  was  divided  into  two  separate  contracts;  the  ripslyn  of 
the  CCN  communications  substrate,  and  its  Implemfintatlon.  This 
report  is  the  final  report  for  the  CCN  implementation  contract, 
N00039-79-C-0482.  For  a  complete  picture  of  BBN's  efforts  in  the 
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design  and  implementation  of  the  Command  Center  Network 
communications  substrate,  it  should  be  read  in  conjunction  with 
BBN  Report  5046,  "Design  of  the  Command  Center  Network:  Final 
Report."  We  recommend  that  the  reader  completely  unfamiliar  with 
the  Command  Center  Network  read  BBN  Report  5046  prior  to  reading 
this  report. 

A  bibliography  of  all  reports  submitted  under  both  CCN 
contracts  may  be  found  at  the  end  of  this  report. 
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2  CCN  IMPLEMENTATION  EXPERIENCES 

The  CCN  communications  substrate  design  and  implementation 
efforts  at  BBN  were  conducted  essentially  concurrently,  resulting 
in  considerable  interaction  between  the  two.  Since 
implementation  of  various  aspects  of  the  CCN  communications 
substrate  was  begun  virtually  as  soon  as  preliminary 
specifications  for  them  were  complete,  it  was  possible  for  early 
implementation  experiences  to  provide  feedback  useful  in 
developing  the  final  specifications  for  the  communications 
substrate.  This  was  particularly  helpful  because  the  application 
of  local  area  network  technology  was  a  relatively  new  and 
uncharted  field  in  late  1979  when  the  CCN  implementatioin  effort 
was  begun, 

2,1  CHAOSNET  As  the  Prototype  CCN  Local  Network 

Of  course,  proceeding  with  the  implementation  while  the 
design  effort  was  still  in  progress  did  lead  to  some  false  steps 
that  later  had  to  be  retraced.  The  largest  such  ”false  step”  was 
the  implementation  of  the  CHAOSNET  hardware  and  software 
originally  selected  for  the  communications  substrate  in  October 
1979.  As  we  described  in  BBN  Report  5046,  by  spring  of  1980  we 
realized  that  the  Net/One  system,  soon  to  be  available  from 
Ungermann-Bass,  Inc.,  would  better  meet  the  objectives  of  CCN, 
Because  implementation  of  much  of  the  CHAOSNET  hardware  and 
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software  was  nearly  complete,  it  was  decided  that  the  best  course 
of  action  was  to  continue  with  the  CHAOSNET  development  and  to 
install  it  in  the  CCN  Testbed  at  NOSC  as  a  prototype  system  so 
that  NOSC  personnel  could  obtain  early  experience  in  developing 
applications  software  for  the  CCN  front-end  processors. 

The  decision  not  to  use  the  CHAOSNET  equipment  for  the 
final,  installed,  CCN  communications  substrate  also  removed  the 
possibility  that  the  Command  Center  Network  would  be  dependent 
upon  equipment  that  was  not  readily  available  and  for  which  there 
was  little  engineering  expertise.  The  decision  to  use  the 
already-developed  CHAOSNET  hardware  and  software  as  a  prototype 
system  to  aid  software  development  turned  the  potential  false 
step  to  advantage. 

The  CHAOSNET  hardware  is  described  in  BBN  Report  4311,  "The 
Command  Center  Network:  Preliminary  Specification,"  Our 
experience  in  implementing  it,  based  on  designs  provided  by  its 
developer,  the  MIT  Artificial  Intelligence  Laboratory  is  related 
in  BBN  Report  5048,  "Command  Center  Network:  Semiannual  Technical 
Report  No.  1 , " 

2.2  The  Change  to  Net/One 

The  decision  to  convert  from  CHAOSNET  equipment  to 
Ungermann-Bass  Net/One  equipment  was  not  made  lightly  because  we 
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had  a  considerable  investment  in  the  CHAOSNET  hardware.  However, 
it  was  clear  that  the  advantages  of  using  a  commercially- 
supported  product  that  met  the  basic  communication  needs  of  the 
CCN  outweighed  the  schedule  and  cost  advantages  of  continuing  our 
commitment  to  the  CHAOSNET.  The  advantages  we  perceived  for  the 
Net/One  system,  discussed  in  BBN  Report  5046,  were  as  follows: 

1.  The  NOSC  CCN  configuration  could  be  expanded 
readily  by  the  purchase  of  additional  Net/One 
units;  additional  CHAOSNET  controllers  and 
receivers  would  have  had  to  be  constructed 
specially,  when  required. 

2.  Ungermann-Bass  would  provide  repair  services  for 
its  Net/One  hardware. 

3.  CCN  would  be  able  to  take  advantage  of  product 
improvements  and  new  developments  from  Ungermann- 
Bass;  lacking  in-depth  CHAOSNET  expertise,  we  had 
no  plans  for  continued  CHAOSNET  development. 

4.  Because  it  was  commercially  available,  Net/One 
equipment  could  be  procured  easily  for  other  Navy 
applications. 

In  addition,  the  plans  for  the  CCN  Network  Services  Manager 
(NSM)  called  for  the  construction  of  a  direct,  high-bandwidth 
interface  device  to  connect  the  NSM  host  to  the  CCN  local 
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network.  For  the  CHAOSNET,  this  would  require  the  development  of 
a  CHAOSNET  controller  specifically  designed  to  interface  to  the 
selected  NSM  host,  the  BBN  Computer  Corporation  C/70  UNIX  system. 
With  Net/One  we  were  able  to  use  the  Net/One  Network  Interface 
Unit  (NIU)  as  a  building  block,  and  design  an  interface 
compatible  with  the  NIU’s  I/O  structure.  This  approach  was  more 
in  keeping  with  the  CCN  objective  of  making  maximum  feasible  use 
of  available  local  network  "building  blocks"  —  and  was  also 
considerably  easier. 


2.3  Implementing  the  LSI-11  Ungermann-Bass  Parallel  Interface 

The  switch  from  CHAOSNET  to  Net/One  was  slightly  less 
satisfactory,  from  an  implementation  point  of  view,  for  the  CCN 
LSI-11  front-end,  gateway,  and  terminal  interface  unit 
processors,  than  it  was  for  the  NSM.  The  CHAOSNET  controllers 
were  UNIBUS  peripheral  devices  (see  BBN  Report  5048);  through  the 
use  of  a  QBUS-to-UNIBUS  converter  (we  selected  the  Able  Computer 
Technology  "QNIVERTER" ) ,  the  CHAOSNET  controller  could  be 
connected  to  the  CCN  LSI-11 »s.  No  such  "direct"  LSI-11  interface 
existed  for  Net/One;  we  had  the  choice  of  either  devel oping ^ 
ourselves,  a  direct  interface  to  the  Net/One  Databus  for  the 
LSI-11,  similar  to  the  interface  we  designed  for  the  C/70,  or  to 
■3.d.apt  a  standard  Net/One  interface  for  our  use. 
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In  this  case,  implementation  considerations  fed  back  to  the 
design  of  the  communications  substrate:  we  concluded  that  the 
eight-bit  parallel,  programmed  I/O  interface  offered  for  the 
Net/One  NIU  would  provide  adequate  performance  for  an  individual 
CCN  LSI-ll  processor,  and  that  the  development  of  a  specialized 
interface  for  the  LSI-11,  roughly  equal  in  complexity  to  the 
interface  for  the  C/70,  would  therefore  not  be  worthwhile.  The 
parallel  programmed  I/O  interface  could  readily  be  connected  to 
the  LSI-11  QBUS  using  a  small  amount  of  interface  circuitry  built 
upon  a  standard  Digital  Equipment  Corporation  (DEC)  LSI-11 
Interface  Foundation  Module,  the  DRV11-P.  The  interface 
constructed  in  this  manner  was  named  the  "LSI-11  Ungermann-Bass 
Parallel  Interface"  (LUBPI). 

Why  the  difference  in  approach  between  the  C/70  and  the 
LSI-11 ‘s?  There  were  two  reasons:  functionality  and  cost.  The 
CCN  LSI-11 ’s  each  served  a  single  purpose;  the  front-end  LSI- 
11  's,  in  particular,  each  served  a  single  command  center 
applications  host.  The  NSM  C/70,  on  the  other  hand,  provided 
services  that  were  shared  by  the  other  hosts  on  the  network.  In 
addition,  the  Network  Services  Manager  design  included  a 
monitoring  capability  that  requires  a  particularly  high-bandwidth 
path  to  the  CCN  local  network.  The  Command  Center  Network  had 
but  one  Network  Services  Manager;  on  the  other  hand,  there  was  a 
multiplicity  of  LSI-11  systems.  From  a  financial  standpoint,  a 
single  copy  of  a  complex  interface  is  more  affordable  than 
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several  copies  would  be. 

The  "protocol"  for  use  of  the  eight-bit  parallel  interface 
for  the  LSI-11 >s  was  devised  with  the  aid  of  Ungermann-Bass,  as 
the  Net/One  standard  product  did  not  make  use  of  that  interface 
at  the  time.  In  fact,  the  only  standard  interface  to  the  Net/One 
NIU,  at  the  time  the  product  was  first  introduced,  was  the  serial 
RS-232  interface,  for  which  Ungermann-Bass  provided  only 
terminal-to-terminal  port,  "virtual  circuit"  service.  Use  of  the 
parallel  port  was  therefore  an  innovation  for  Ungermann-Bass  as 
well  as  for  us. 

The  parallel  port  protocol  specified  in  BBN  Report  ^991, 
"Design  of  the  Command  Center  Network:  Final  Hardware  and 
Software  Specifications,"  was  developed  in  a  very  pragmatic  way. 
The  parallel  port  of  the  NIU  was  based  on  a  Zilog  Z80  parallel 
I/O  (PIO)  integrated  circuit.  The  Z80  PIO  permits  several  forms 
of  communication,  including  use  as  a  dual  port  with  limited 
"hand-shaking"  for  data  transfer,  or  as  a  single  port  with 
separate  control  and  data  bit  signals.  Because  we  wished  to 
exchange  interrupt  information  as  well  as  data,  we  chose  the  mode 
that  provided  the  single  port  with  control  signals. 

With  that  decision  made,  we  then  faced  the  problem  of 
connecting  the  parallel  port,  so  configured,  to  the  LSI-11. 
Here,  we  had  somewhat  less  of  a  choice.  The  interface  signals  we 
could  obtain  from  the  Z80  PIO  port  were  almost  compatible  with 
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the  interface  signals  from  one  standard  DEC  parallel  interface 
module,  the  DRV11,  However,  "almost"  did  not  permit  a  direct 
interconnection,  and  no  other  DEC  parallel  interface  module  came 
as  close.  The  DRVIl-P  Interface  Foundation  Module,  on  the  other 
hand,  offered  the  opportunity  for  us  to  build  a  parallel 
interface  to  almost  any  interface  specification  necessary,  with 
the  LSI-11  QBUS  logic  handled  by  the  circuitry  of  the  DRV11-P. 
The  interface  we  required  was,  of  course,  very  close  to  the  DRV11 
specification;  we  were  able  to  implement  it  on  the  DRV11-P  by 
using  only  six  integrated  circuit  packages.  The  LUBPI  could  be 
termed  a  "semi-custom"  interface;  from  the  point  of  view  of 
designing,  implementing,  and  constructing  the  required  copies  of 
the  interface,  it  was  a  far  better  approach  than  to  design  either 
an  LSI-11  interface  or  a  Net/One  interface  from  scratch. 


2.4  The  C/70  Network  Services  Manager  Interface  for  Net/One 

Because  of  the  importance  and  nature  of  the  role  the  C/70 
Network  Services  Manager  plays  in  the  Command  Center  Network,  the 
C/70  must  be  attached  to  the  Net/One  local  network  system  via  a 
direct,  high-bandwidth  interface.  While  the  architecture  of  the 
interface  device,  called  the  "MBB  Ungermann-Bass  Interface" 
(MUBI),  was  determined  by  the  relationship  between  the  I/O 
architecture  of  the  Ungermann-Bass  Net/One  processor  and  the  C/70 
I/O  bus,  its  physical  form  was  strongly  influenced  by 
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implementation  considerations.  Two  primary  facts  dictated  the 
shape  taken  by  the  MUBI: 

1.  The  C/70  I/O  bus  could  be  extended,  via  a  set  of 
three  100-conductor  flat  ribbon  cables,  to  a 
length  of  six  to  ten  feet. 

2.  The  Net/One  Network  Interface  Unit  Databus  was 
designed  specifically  to  be  internal  to  the 
Net/One  NIU. 

Clearly,  the  only  configuration  possible  was  one  that  placed 
the  MUBI  inside  the  Net/One  NIU  enclosure,  powered  from  the  NIU 
and  cabled  to  the  C/70.  This  is  the  form  the  MUBI  took. 

Our  experiences  in  implementing  the  MUBI  hardware  and 
software  are  described  in  BBN  Report  5050,  "Command  Center 
Network:  Semiannual  Technical  Report  No.  3." 

2.5  LSI-11  System  Software 

The  LSI-11  system  software  we  used  in  the  CCN  project  was, 
for  the  most  part,  obtained  from  other  sources,  then  adapted  and 
integrated  with  software  written  specifically  for  CCN.  The 
version  of  the  MOS  operating  system  we  used  was  derived  from  the 
SRI  International  MOS  system,  and  was  used  in  several  DARPA- 
sponsored  networking  projects  at  BBN.  The  TCP,  IP,  and  TIU 
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implementations  were  obtained  from  the  same  source. 

The  local  network  device  driver  software  for  both  CHAOSNET 
and  Net/One  was,  of  course,  developed  specifically  for  CCN.  In 
addition,  the  CCN-ARPANET  gateway  code  was  written  specifically 
for  CCN,  as  there  was  no  suitable  "standard"  Internet  LSI-11 
gateway  at  the  time  the  CCN-ARPANET  gateway  was  developed. 

2.6  LSI-11  Applications  Software  Experiences 

In  CCN,  the  term  "applications  software"  refers  to  software 
written  for  the  LSI-11  front-end  systems  that  deals  with  the 
interface  to  the  attached  command  center  applications  processor. 
Written  by  NOSC  personnel,  and  integrated  with  the  BBN-provided 
LSI-11  systems  software  as  part  of  the  CCN  installation  effort, 
this  software  varied  in  size,  complexity,  and  functionality  from 
application  to  application.  The  most  complex  front-end 
applications  software,  the  NAVMACS  message  processor  server,  was 
a  particularly  "tight  fit"  within  the  28  Kword  memory  of  the  CCN 
LSI-11 »s.  The  ability  to  fit  this  software  into  the  CCN  front- 
end,  along  with  the  MOS  operating  system,  the  Net/One  device 
driver,  and  TCP/IP  process,  required  a  tradeoff  that  involved 
reducing  the  size  and  number  of  TCP/IP  packet  buffers  in  the 
front-end;  this,  in  turn,  had  a  direct  bearing  on  the  performance 
of  the  CCN  NAVMACS  application. 
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The  conclusion  to  be  drawn  from  this  experience  is  that  the 
28  Kword  LSI-11 ’s  are  not  well  suited  to  a  role  in  the  CCN  that 
involves  a  considerable  amount  of  applications  software  in  the 
front-end  processor.  Unfortunately,  conversion  to  the  LSI-11/23, 
with  extended  memory  capabilities,  is  not  a  reasonable  solution, 
as  MOS  relies  extensively  on  single-word  pointers  and  cannot  take 

't- 

advantage  of  the  extended  memory.  There  is,  therefore,  no 
short-term  solution  to  this  problem.  There  are,  however,  two 
possible  long-range  solutions: 

1.  Move  most  of  the  application  processing  to  another 
CCN  node,  leaving  little  functionality  beyond 
mediation  between  the  application  processor 
interface  and  TCP,  in  the  front-end.  The  Network 
Services  Manager  would  be  a  suitable  node  for 
processing  that  does  not  have  a  strict  real-time 
requirement.  There  is  no  node  in  the  current  CCN 
configuration  that  can  handle  real-time  processing 
in  this  manner.  Such  a  node,  which  we  call  a 
"Network  Traffic  Processor,"  should  provide: 

a.  a  small,  real-time  operating  system  (similar 
to  MOS), 

b.  1/4  -  1/2  MByte  of  memory,  and 

c.  an  I/O  structure  that  can  readily  accommodate 
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a  high-bandwidth  interface  to  the  CCN  local 
area  network. 

2.  Replace  the  LSI-11  front-end  processor  with  a 
microcomputer  system  based  on  a  more  modern  and 
more  capable  microprocessor  such  as  the  Motorola 
68000  (with  attributes  similar  to  those  described 
in  a.  and  b.  of  solution  1.)  which  could  properly 
handle  the  traffic  processing  load. 

Since  the  microcomputer  systems  proposed  in  the  two 
solutions  are  in  fact  quite  similari  the  issue  really  comes  down 
to  the  question: 

"Should  Inaffic  processing  in  CCN  be  performed  in  the 
front-end,  or  should  the  functionality  of  the  front-end 
be  limited  to  communication  functions  and  the  handling 
of  the  application  processor  interface?" 

At  present  this  question  remains  unanswered  and  it  is  an 
interesting  topic  for  future  study. 


2.7  C/70  Network  Services  Manager  Software 

Software  for  the  C/70  Network  Services  Manager  is  the  one 
area  in  which  the  system  delivered  to  NOSC  for  the  CCN  Testbed 
diverged  from  the  specifications  developed  under  the  CCN  design 
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effort.  This  was  due  largely  to  schedule  considerations:  TCP, 
in  particular,  was  not  available  for  the  C/70  UNIX  system  at  the 
time  it  would  have  been  required  for  the  NSM.  Therefore,  the 
software  provided  for  the  CCN  Testbed  was  implemented  so  as  not 
to  require  TCP  (for  example,  we  provided  an  implementation  of  the 
"Trivial  File  Transfer  Protocol"  to  effect  file  transfers  between 
the  ACCAT  PDP-11/70  UNIX  system  and  the  C/70  NSM  via  the  ARPANET 
and  the  CCN-ARPANET  gateway). 

In  addition,  as  we  described  in  BBN  Report  5046,  the  initial 
decision  made  to  adapt  the  NU  network  monitoring  and  control 
system  software  for  use  with  CCN  had  to  be  rescinded  when  it 
became  apparent  that  NU  would  not  be  completed  in  time,  and, 
moreover,  that  the  modifications  required  to  NU  would  be 
sufficiently  extensive  that,  to  meet  the  CCN  project  schedule,  an 
alternative  course  of  action  was  preferable.  This  led  to  the 
development  of  the  CCN  Monitoring  System  software  which  is 
described  in  BBN  Report  49'91 . 

One  particularly  bright  spot  in  the  Network  Services  Manager 
software  picture  was  the  microcode  and  macrocode  UNIX  kernel 
device  driver  software  developed  for  the  MUBI.  As  described  in 
BBN  Report  5051 ,  this  software,  in  conjunction  with  the  MUBI 
hardware,  was  demonstrated  to  provide  a  raw  packet  throughput 
rate  of  approximately  2  Mb/s,  roughly  half  the  aggregate 
bandwidth  available  on  the  Ungermann-Bass  Net/One  4  Mb/s 
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contention  bus  coaxial  cable  network.  This  successful  experiment 
helped  to  validate  the  design  philosophy  of  the  software, 
described  in  BBN  Report  504?,  which  was  to  provide  a  high- 
bandwidth  path  through  the  UNIX  kernel  between  the  MUBI  hardware 
and  user-level  code. 
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3  OBSERVATIONS  AND  CONCLUSIONS 

The  Command  Center  Network  implementation  effort  may  be 
viewed  on  the  one  hand,  as  a  validation  of  the  network  design  and 
specifications  developed  during  the  CCN  design  effort,  and  on  the 
other,  as  a  laboratory  experiment  in  the  application  of  local 
area  network  technology  to  a  practical  problem.  In  fact,  the  CCN 
implementation  effort  was  both  of  these;  it  produced  an 
"exploratory  development  model"  of  the  Command  Center  Network  for 
use  at  the  CCN  Testbed  at  NOSC,  and  it  provided  an  interesting 
set  of  experiences  that,  we  hope,  can  guide  others  in  applying 
local  network  technology. 

Perhaps  the  most  significant  conclusion  to  draw  specifically 
from  the  CCN  implementation  experience  is  that  selecting  and 
procuring  the  local  area  network  technology  to  be  used  in  an 
application  such  as  CCN  is,  truly,  the  tip  of  the  proverbial 
iceberg.  This  lesson  has  been  learned  in  applying  long-haul 
network  technology;  but  it  seems  it  must  be  learned  again  with 
local  network  technology.  The  trap  is  this:  local  network 
technology  is,  by  its  very  nature,  inexpensive  when  compared  to 
long-haul  network  technology.  Indeed,  it  is  this  property  of 
local  network  technology  that  makes  it  possible  to  contemplate 
local  networks  with  hosts  that  are  microcomputer  systems, 
concentrators  for  small  numbers  of  terminals,  etc.  We  therefore 
Mant  to  believe  that  the  software  required  for  local  area 
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networks  should  be  similarly  "inexpensive"  and  less  complex  than 
the  software  needed  for  applications  of  long-haul  networking. 
Experience  shows  that  it  is  not.  There  are  a  number  reasons  for 
this  and  we  present  two  that  we  feel  are  particularly  important: 

1.  Local  area  networks  rarely  stand  alone.  While  one 
might  contemplate  the  use  of  simplified  protocols 
that  take  advantage  of  the  high  throughput,  low 
delay  characteristics  of  local  area  networks,  the 
need  to  intercommunicate  with  other  networks  on  a 
regular  basis  argues  against  this  approach. 

2.  The  high  bandwidth  of  local  area  networks  can 
present  problems  to  the  host  programmer.  Buffer 
allocation  strategies  that  may  have  sufficed  when 
one  was  dealing  with  a  9.6  Kb/s  serial  line  or  a 
100  Kb/s  network  attachment  stand  a  good  chance  of 
failing  when  applied  to  a  10  Mb/s  local  area 
network. 

With  the  second  point  above  we  do  not  mean  to  imply  that  the 

Mb/s  capacity  of  the  local  area  network  will 

necessarily  be  "aimed"  at  a  single  host  for  a  long  period  of  time 
(although  this  is  conceptually  possible,  we  believe  it  is  a 
pathological  case  that  need  not  be  handled  particularly  well)j 
however,  in  order  to  take  advantage  of  the  local  area  network, 

the  host  must  be  prepared  to  receive  a  sequence  or  burst  of 
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packets  from  the  network  in  rapid  succession.  By  way  of  analogy, 
if  the  problem  of  handling  a  long-haul  network  communications 
interface  is  viewed  as  drinking  from  a  water  fountain,  the 
problem  of  interfacing  to  a  local  area  network  may  be  viewed  as 
taking  a  drink  from  a  firehose;  the  same  commodity  is  being 
obtained,  but  there's  something  qualitatively  different  about  the 
experience. 

One  final  issue  raised  in  the  course  of  the  implementation 
of  the  Command  Center  Network  concerns  the  role  of  the  CCN 
front-end  processors,  and  where  "traffic  processing"  can  best  be 
performed.  We  believe  that  this  issue,  discussed  in  section  2,6, 
is  crucial  to  the  long-term  success  of  the  CCN  and  should  be 
addressed  in  any  future  work  with  the  Command  Center  Network. 
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